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(57) Abstract: A system, method, and article of manufacture are provided for navigating an electronic data source that has a scripted 
onhne mterface by means of spoken natural language. When a spoken natural language input request is received from a user it is 
interpreted. A navigation query is constructed based on the interpreution of the speech input and a template extracted by scraoine an 
onhne scnpted interface to the data source. The resulting interpretation of the request is theieopon used to automatically constfuct 
an operational navigation query to retrieve the desired information from one or more electronic network data sources which is then 
transmitted to a client device of the user. 
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Accessing Network-Based Electronic Information Through Scripted 
Online Interfaces Using Spoken Natural Language Input 

Background of the Invention 

5 This is a Continuation In Part ofco-pending U.S. Patent Application No. 

09/225,198, filed January 5, 1999, Provisional U.S. Patent Application No. 
60/124,718, filed March 17, 1999, Provisional U.S. Patent Application No. 
60/124,720, filed March 17, 1999, and Provisional U.S. Patent Application No. 
60/124,719, filed March 17, 1999, firom which applications priority is claimed and 
1 0 these application are incorporated herein by reference. 

The present invention relates generally to the navigation of electronic data by 
means of spoken natural language requests, and to feedback mechanisms and methods 
for resolving the errors and ambiguities that may be associated with such requests. 
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As global electronic connectivity continues to grow, and the universe of 
electronic data potentially available to users continues to expand, there is a growing 
need for information navigation technology that allows relatively naive users to 
navigate and access desired data by means of natural language input. In many of the 
most important markets - including the home entertainment arena, as well as mobile 
computing - spoken natiiral language input is highly desirable, if not ideal. As just 
one example, the proUferation of high-bandwidth communications infi-astinicture for 
the home entertainment market (cable, satelhte, broadband) enables delivery of 
movies-on-demand and other interactive multimedia content to the consumer's home 
television set. For users to take full advantage of tiiis content sb-eam ultimately 
requires interactive navigation of content databases in a manner that is too complex 
for user-fiiendly selection by means of a traditional remote-conti-ol choker. Allowing 
spoken natiu-al language requests as tiie input modality for rapidly searching and 
accessing desired content is an important objective for a successful consumer 
entertainment product in a context offering a dizzying range of database content 
choices. As further examples, this same need to drive navigation of (and transaction 
with) relatively complex data warehoifees using spoken natiiral language requests 
appUes equaUy to surfing the IhtemetAVeb or other networks for general information, 
multimedia content, or e-commerce transactions. 
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In general, the existing navigational systems for browsing electronic 
databases and data warehouses (search engines, menus, etc.), have been designed 
without navigation via spoken natural language as a specific goal. So today's world is 
fiill of existing electronic data navigation systems that do not assume browsing via 
5 natural spoken commands, but rather assume text and mouse-click inputs (or in the 
case of TV remote controls, even less). Simply recognizing voice commands within 
an extremely limited vocabulary and grammar ~ the spoken equivalent of button/cUck 
input (e.g., speaking "channel 5" selects TV channel 5) - is really not sufficient by 
itself to satisfy the objectives described above, hi order to deliver a true "win" for 

10 users, the voice-driven fi^ont-end must accept spoken natural language input in a 
manner that is intuitive to users. For example, the firont-end should not require 
learning a highly specialized command language or format. More fundamentally, the 
firont-axd must allow users to speak directly in terms of what the user ultimately wants 
- e.g., "I'd like to see a Western fihn directed by Clint Eastwood" ~ as opposed to 

1 5 speaking in terms of arbitrary navigation stmctures (e.g., hierarchical layers of menus, 
commands, etc.) that are essentiailly artifacts reflecting constraints of the pre-existing 
text/click navigation system. At the same time, the fi-ont-end must recognize and 
accommodate the reality that a stream of naive spoken natural language input will, 
over time, typically present a variety of errors and/or ambiguities: e.g., 

20 garbled/unrecognized words (did the user say "Eastwood" or "Easter"?) and under- 
constrained requests ("Show me the Clint Eastwood movie"). An approach is needed 
for handling and resolving such errors and ambiguities in a rapid, user-fiiendly, non- 
finstrating maimer. 

What is needed is a methodology and apparatus for rapidly constructing a 
25 voice-driven fi-ont-end atop an existing, non-voice data navigation system, whereby 
users can interact by means of intuitive natural language input not strictly conforming 
to the step-by-step browsing architecture of the existing navigation system, and 
wherein any errors or ambiguities in user input are rapidly and conveniently resolved. 
The solution to this need should be compatible wilh the constraints of a multi-user, 
30 distributed environment such as the hitemetAVeb or a proprietary high-bandwidth 

content delivery network; a solution contemplating one-at-a-time user interactions at a 
single location is insufficient, for example. 
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Summary of the Invention 

The present invention addresses the above needs by providing a system, 
method, and article of manufacture for navigating network-based electronic 
multimedia data sources that contain multimedia content in response to spoken ML 
input requests. When a spoken natural language input request is received from a user, 
it is interpreted, such as by using a speech recognition engine to extract speech data 
from acoustic voice signals, and using a natural language parser to linguistically parse 
the speech data. The interpretation of the spoken natural language request can be 
perfonned on a computing device locally with the user or remotely from the user. A 
navigation query is constructed based on the interpretation of the speech input and a 
template extracted by scraping an online scripted interface to the data source. The 
resulting interpretation of the request is thereupon used to automatically construct an 
operational navigation query to retrieve the desired information from one or more 
electronic network data sources, which is then transmitted to a client device of the 
user. 

Typically, errors or ambiguities emerge in the interpretation of the spoken ML 
request, such that the system cannot instantiate a complete, valid navigational 
template. This is to be expected occasionally, and one preferred aspect of the 
invention is the ability to handle such errors and ambiguities in relatively graceftd and 
user-friendly manner. Instead of simply rejecting such input and defaulting to 
traditional input modes or simply asking the user to try again, a preferred embodiment 
of the present invention seeks to converge rapidly toward instantiation of a valid 
navigational template by soliciting additional clarification from the user as necessary, 
either before or after a navigation of the data source, via multimodal input, i.e., by 
means of menu selection or other input modalities including and in addition to spoken 
natural language. This clarifying, multi-modal dialogue takes advantage of whatever 
partial navigational information has been gleaned from the initial interpretation of the 
user's spoken NL request. This clarification process continues until the system 
converges toward an adequately instantiated navigational template, which is in tiim 
used to navigate tiie network-based data and retrieve the user's desired information. 
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The retrieved information is transmitted 
on a suitable client display device. 
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Brief Description of the Drawings 

The invention, together with further advantages thereof, may best be 
understood by reference to the following description taken in conjunction with the 
accompanying drawings in which: 

Figure la illustrates a system providing a spoken natural language interface 
for network-based information navigation, in accordance with an embodiment of the 
present invention with server-side processing of requests; 

Figure lb illustrates another system providing a spoken natural language 
interface for network-based information navigation, in accordance with an 
embodiment of the present invention with client-side processing of requests; 

Figure 2 illustrates a system providing a spokai natural language interface for 
network-based information navigation, in accordance with an embodiment of the 
present invention for a mobile computing scenario; 

Figure 3 illustrates the functional logic components of a request processing 
module in accordance with an embodiment of the presait invention; 

Figure 4 illustrates a process utiUzing spoken natural language for navigating 
an electronic database in accordance with one embodiment of the presait invention; 

Figure 5 illustrates a process for consh^cting a navigational query for 
accessing an online data source via an interactive, scripted (e.g., CGI) form; and 

Figure 6 illustiates an embodiment of the present invention utilizing a 
community of distiibuted, collaborating electronic agents. 
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Detailed Description of the Invention 
1. System Architecture 

a. Server-End Processing of Spoken Input 

Figure la is an illustration of a data navigation system driven by spoken 
natural language input, in accordance with one embodiment of the present invention. 
As shown, a user's voice input data is captured by a voice input device 102, such as a 
microphone. Preferably voice input device 102 includes a button or the like that can 
be pressed or held-dovm to activate a listening mode, so that tiie system need not 
continually pay attention to, or be confused by, irrelevant background noise. In one 
preferred embodiment well-suited for the home entertainment setting, voice input 
device 102 is a portable remote control device with an integrated microphone, and the 
voice data is transmitted from device 102 preferably via infrared (or other wireless) 
link to conununications box 104 (e.g., a set-top box or a similar communications 
device that is capable of retransmitting the raw voice data and/or processing the voice 
data) local to the user's environment and coupled to communications network 106. 
The voice data is then transmitted across network 106 to a remote server or servers 
108. The voice data may preferably be transmitted in compressed digitized form, or 
alternatively -particularly whrare bandwidth constraints are significant- in analog 
format (e.g., via frequency modulated transmission), in the latter case being digitized 
upon arrival at remote servo: 108. 

At remote server 108, the voice data is processed by request processing logic 
300 in order to understand the user's request and constiixct an appropriate query or 
request for navigation of remote data source 1 10, in accordance with the interpretation 
process exempUfied in Figure 4 and Figure 5 and discussed in greater detail below. 
For purposes of executing this process, request processing logic 300 comprises 
functional modules including speech recognition engine 310, natural language (NL) 
parser 320, query construction logic 330, and query refinement logic 340, as shown in 
Figure 3. Data source 1 10 may comprise database(s), Intemet/web site(s), or other 
electronic information repositories, and preferably resides on a centi-al server or 
servers - which may or may not be tiie same as server 108, depending on the storage 
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and bandwidth needs of the application and the resources available to the practitioner. 
Data source 110 may include multimedia content, such as movies or other digital 
video and audio content, other various forms of entertainment data, or other electronic 
information. The contents of data source 110 are navigated ~ i.e., the contents are 
5 accessed and searched, for retrieval of the particular information desired by the user ~ 
using the processes of Figures 4 and 5 as described in greater detail below. 

Once the desired information has been retrieved from data sovirce 110, it is 
electronicaUy transmitted via network 106 to the user for viewing on client display 
device 1 12. In a preferred embodiment well-suited for the home entertainment setting, 

10 display device 1 12 is a television monitor or similar audiovisual entertainment device, 
typically in stationary position for comfortable viewing by users. In addition, in such 
preferred embodiment, display device 112 is coupled to or integrated with a 
communications box (which is preferably the same as communications box 104, but 
may also be a separate unit) for receiving and decoding/fomiatting the desired 

1 5 electronic information that is received across communications network 1 06. 

Network 106 is a two-way electronic commimications network and may be 
embodied in electronic communication infrastmcture including coaxial (cable 
television) lines, DSL, fiber-optic cable, traditional copper wire (twisted pair), or any 
other type of hardwired connection. Network 106 may also include a wireless 
20 connection such as a sateUite-based connection, cellular connection, or other type of 
wireless connection. Network 106 may be part of the Internet and may support 
TCP/IP communications, or may be embodied in a proprietary network, or in any 
other electronic communications network infrastructure, whether packet-switched or 
connection-oriented. A design consideration is that network 106 preferably provide 
suitable bandwidth depending upon the nature of the content anticipated for the 
desired application. 

b. Client-End Pm r-essine of Sr?oken fopnt 

Figure lb is an iUustration of a data navigation system driven by spoken 
natural language input, in accordance with a second embodiment of the present 
invention. Again, a user's voice input data is c^tured by a voice input device 102, 
such as a microphone. In the embodiment shown in Figure lb. the voice data is 
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transmitted from device 202 to requests processing logic 300, hosted on a local speech 
processor, for processing and interpretation. In the preferred embodiment illustrated 
in Figure lb, the local speech processor is conveniently integrated as part of 
cormnunications box 104, although implementation in a physically separate (but 
5 communicatively coupled) unit is also possible as will be readily apparent to those of 
skill in the art. The voice data is processed by the components of request processing 
logic 300 in order to understand the user's request and construct an appropriate query 
or request for navigation of remote data source 110, in accordance with the 
interpretation process exemplified in Figures 4 and 5 as discussed in greater detail 
10 below. 

The resulting navigational query is then transmitted electronically across 
network 106 to data source 110, which preferably resides on a central server or 
servers 108. As in Figure la, data source 110 may comprise database(s), hitemet/web 
site(s), or other electronic information repositories, and preferably may include 
15 multimedia content, such as movies or other digital video and audio content, other 
various forms of entertainment data, or other electronic information. The contents of 
data source 1 10 are then navigated - i.e., the contents are accessed and searched, for 
retrieval of the particular information desired by the user - preferably using the 
process of Figures 4 and 5 as described m greater detail below. Once the desired 
20 information has been retrieved from data source 110, it is electronically transmitted 
via network 1 06 to the user for viewing on client display device 112. 

In one embodiment in accordance witli Figure lb and well-suited for the home 
entertamment setting, voice input device 102 is a portable remote control device with 
an integrated microphone, and the voice data is transmitted from device 102 
preferably via infi^red (or other wireless) link to the local speech processor. The 
local speech, processor is coupled to communications network 106, and also 
preferably to client display device 112 (especially for purposes of query refinement 
transmissions, as discussed below in connection with Figure 4, step 412), and 
preferably may be integrated within or coupled to communications box 104, Txi 
addition, especially for purposes of a home entertainment apphcation, display device 
1 12 is preferably a television monitor or sinwlar audiovisual entertainment device, 
typically in stationary position for comfortable viewing by users. In addition, in such 
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preferred embodiment, display device 1 12 is coupled to a communications box (which 
is preferably the same as communications box 104, but may also be a physically 
separate unit) for receiving and decoding/formatting the desired electronic 
information that is received across communications network 106. 

Design considerations favoring server-side processing and interpretation of 
spoken input requests, as exempUfied in Figure la, include minimizing the need to 
distribute costly computational hardware and software to all client users in order to 
perform speech and language processing. Design considerations favoring client-side 
processing, as exemplified in Figure lb, include minimizing the quantity of data sent 
upstream across the network from each client, as the speech recognition is performed 
before transmission across the network and only the query data and/or request needs 
to be sent, thus reducing the upstream bandwidth requirements. 

c. Mobile Client Embodiment 

A mobile computing embodiment of the present invention may be 
implemented by practitioners as a variation on fee embodiments of either Figure la or 
Figure lb. For example, as depicted in Figure 2, a mobile variation in accordance 
with the server-side processing architecture illustrated in Figure la may be 
implemented by replacing voice input device 102, communications box 104, and 
client display device 1 12, with an integrated, mobile, information appliance 202 such 
as a cellular telephone or wireless personal digital assistant (wireless PDA). Mobile 
information appUance 202 essentially performs the functions of the replaced 
components. Thus, mobile information appliance 202 receives spoken natural 
language input requests from the user in the form of voice data, and transmits that 
data (preferably via wireless data receiving station 204) across communications 
network 206 for server-side interpretation of the request, in similar fashion as 
described above in connection with Figure 1. Navigation of data source 210 and 
retrieval of desired information likewise proceeds in an analogous manner as 
described above. Display information transmitted electronically back to the user 
across network 206 is displayed for the user on the display of information appliance 
202, and audio infomiation is output through the apphance's speakere. 
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Practitioners will further appreciate, in light of the above teachings, that if 
mobile information jqjpliance 202 is equipped with sufficient computational 
processing powar, then a mobile variation of the client-side architecture exemplified 
in Figure 2 may similarly be inq>Iemented. In that case, the modules corresponding to 
5 request processing logic 300 would be embodied locally in the computational 
resources of mobile information apphance 202, and the logical flow of data would 
otherwise follow in a manner analogous to that previously described in connection 
with Figure lb. 

As illustrated in Figure 2, multiple users, each having their own cUent input 
10 device, may issue requests, simultaneously or otherwise, for navigation of data source 
210. This is equally true (though not explicitly drawn) for the embodiments depicted 
in Figures la and lb. Data source 210 (or 100), being a network accessible 
information resource, has typically akeady been constracted to support access 
requests from simultaneous multiple network users, as known by practitioners of 
15 ordinary skill in the art. In the case of server-side speech processing, as exemplified 
in Figures la and 2, the interpretation logic and error correction logic modules are 
also preferably designed and implemented to support queuing and multi-tasking of 
requests from multiple simultaneous network users, as will be j^preciated by those of 
skill in the art. 

20 It will be apparent to those skilled in the art that additional implementations, 

permutations and combinations of the embodiments set forth in Figures la, lb, and 2 
may be created without straying from the scope and spirit of the present invention. 
For example, practitioners wiU understand, in light of the above teachings and design 
considerations, that it is possible to divide and allocate the functional components of 

>5 request processing logic 300 between cHent and server. For example, speech 
recognition ~ in entirety, or perhaps just early stages such as feature extraction ~ 
might be perfonned locaUy on the client end, perhaps to reduce bandwidth 
requirements, while natural language parsing and other necessary processing might be 
performed upstream on the server end, so that more extensive computational power 

10 need not be distributed locally to each cUent. In that case, corresponding portions of 
request processing logic 300, such as speech recognition engine 310 or portions 
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thereof, would reside locally at the client as in Figure lb, while other component 
modules would be hosted at the server end as in Figures la and 2. 

Further, practitioners may choose to implement the each of the various 
embodiments described above on any number of different hardware and software 
computing platforms and environments and various combinations thereof, including, 
by way of just a few examples: a general-purpose hardware microprocessor such as 
the Intel Pentium series; operating system software such as Microsoft Windows/CE, 
Pahn OS, or Apple Mac OS (particularly for chent devices and client-side 
processing), or Unix, Linux, or Windows/NT (the latter three particularly for network 
data servers and server-side processing), and/or proprietary information access 
platforms such as Microsoft's WebTV or the Diva Systems video-on-demand system. 

2. Processing Methodology 

The present invention provides a spoken natural language interface for 
interrogation of remote electronic databases and retrieval of desired information. A 
preferred embodiment of the present invention utilizes the basic methodology outUned 
in the flow diagram of Figure 4 in order to provide this interface. This methodology 
will now be discussed. 

a. Interpreti ng Spoken Natural Language Requests 

At step 402, the user's spoken request for information is initially received in 
the form of raw (acoustic) voice data by a suitable input device, as previously 
discussed in connection with Figures 1-2. At step 404 the voice data received from 
the user is interpreted in order to understand the user's request for mformation. 
Preferably this step includes performing speech recognition in order to extract words 
from the voice data, and fiirther includes natural language parsing of those words in 
order to generate a structured linguistic representation of the user's request. 

Speech recognition in step 404 is performed using speech recognition engine 
310. A variety of commercial quality, speech recognition engines are readily 
available on the market, as practitioners will know. For example. Nuance 
Communications offers a suite of speech recognition engines, including Nuance 6. its 
current flagship product, and Nuance Express, a lower cost package for entry-level 
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applications. As one other example, IBM offers the ViaVoice speech recognition 
engine, including a low-cost shrink-wrapped version available through popular 
consumer distribution channels. Basically, a speech recognition engine processes 
acoustic voice data and attempts to generate a text stream of recognized words. 

5 Typically, the speech recognition engine is provided with a vocabulary lexicon 

of likely words or phrases that the recognition engine can match against its analysis of 
acoustical signals, for purposes of a given application. Prefa-ably, the lexicon is 
dynamically adjusted to reflect tiie current user context, as established by the 
preceding user inputs. For example, if a user is engaged in a dialogue with the system 

10 about movie selection, the recognition engine's vocabulary may preferably be adjusted 
to favor relevant words and phrases, such as a stored Ust of proper names for popular 
movie actors and directors, etc. Whereas if the current dialogue involves selection 
and viewing of a sports event, the engine's vocabulary might preferably be adjusted to 
favor a stored hst of proper names for professional sports teams, etc. In addition, a 

15 speech recognition engine is provided with language models fliat help the aigine 
predict the most likely interpretation of a given segment of acoustical voice data, in 
the cuirent context of phonemes or words in which the segment iq)peais. In addition, 
speech recognition engines often echo to the user, in more or less real-tune, a 
transcription of the engine's best guess at what the user has said, giving the user an 

20 opporhmity to confirm or reject. 

In a fiutiier aspect of step 404, natural language interpreter (or parser) 320 
hnguistically parses and interprets the textual output of the speech recognition engine. 
In a preferred embodiment of the present invention, the natural-language interpreter 
attempts to determine both the meaning of spoken words (semantic processing) as 
well as the grammar of the statement (syntactic processing), such as the Gemini 
Natural Language Understanding System developed by SRI International. The 
Gemini system is described in detail in publications entitled "Gemini: A Natural 
Language System for Spoken-Language Understanding" and "hiterleaving Syntax and 
Semantics in an Efficient Bottom-Up Parser," both of which ate currently available 
online at http://www.ai.sri.com/natural-langua pe/Droiects/aTDa-sls/nat-lanff html 
(Copies of those pubhcations are also included in an information disclosure statement 
submitted herewith, and are incorporated herein by this reference). Briefly, Gemini 
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applies a set of syntactic and semantic grammar rules to a word string using a bottom- 
up parser to generate a logical form, which is a structured representation of tiie 
context-independent meaning of the string. Gemini can be used with a variety of 
grammars, including general English grammar as well as application-specific 
granmiars. The Gemini parser is based on "unification grammar," meaning that 
grammatical categories incorporate features that can be assigned values; so that when 
grammatical category expressiom are matched in the course of parsing or semantic 
interpretation, the information contained in the features is combined, and if the feature 
values are incompatible the match fails. 

It is possible for some applications to achieve a significant reduction in speech 
recognition error by using the natural-language processing system to re-score 
recognition hypotheses. For example, tlie grammars defined for a language parser 
like Gemini may be compiled into context-free grammar that, in turn, can be used 
directly as language models for speech recognition engines like the Nuance 
recognizer. Further details on this methodology are provided in the pubUcation 
"Combining Linguistic and Statistical Knowledge Sources in Natural-Language 
Processing for ATIS" which is currently available online through 
http://www.ai.sri.com/natural-langu age/proiects/arDa-sls/sp nl-.iTit htiTiT A copy of this 
publication is included in an information disclosure submitted herewith, and is 
incorporated herein by this reference. 

Li an embodiment of the present invention that may be preferable for some 
appUcations. the natural language interpreter 'leams" fi-om the past usage patterns of 
a particular user or of groups of users. In such an embodiment, the successfiilly 
interpreted requests of users are stored, and can then be used to enhance accuracy by 
comparing a current request to the stored requests, thereby allowing selection of a 
most probable result. 

b. Construc ting Navigation Queries 

Li step 405 request processing logic 300 identifies and selects an appropriate 
online data source where the desired information (in this case, current weather reports 
for a given city) can be found. Such selection may involve look-iqj in a locally stored 
table, or possibly dynamic searching through an online search engine, or other online 
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search techniques. For some apphcations, an embodiment of the present invention 
may be implemented in which only access to a particular data source (such as a 
particular vendor's proprietary content database) is supported; in that case, step 405 
may be trivial or may be eliminated entirely. 

5 Step 406 attempts to construct a navigation query, reflecting the mterpretation 

of step 404. This operation is preferably perforaied by query constraction logic 330. 

A "navigation query" means an electronic query, form, series of menu 
selections, or the like; being stmctured appropriately so as to navigate a particular 
data source of interest in search of desired information. In other words, a navigation 
10 query is constructed such that it includes whatever content and structure is required in 
order to access desired information electronically from a particular database or data 
source of interest. 

For example, for many existing electronic databases, a navigation query can 
be embodied using a formal database query language such as Standard Query 

15 Language (SQL). For many databases, a navigation query can be constmcted through 
a more user-friendly interactive front-end, such as a series of menus and/or interactive 
forms to be selected or filled in. SQL is a standard interactive and programming 
language for getting information from and updating a database. SQL is both an ANSI 
and an ISO standard. As is well known to practitioners, a Relational Database 

20 Management System (RDBMS), such as Microsoft's Access, Oracle's Oracle?, and 
Computer Associates' CA-Openlngres, allow programmers to create, update, and 
administer a relational database. Practitioners of ordinary skill in the art will be 
thoroughly famihar with the notion of database navigation through structured query, 
and will be readily able to appreciate and utihze the existing data structures and 

25 navigational mechanisms for a given database, or to create such stmctures and 
mechanisms where desired. 

In accordance with the present invention, the query constructed in step 406 
must reflect the user's request as interpreted by the speech recognition engine and the 
NL parser in step 404. In embodiments of the present invention wherein data source 
30 1 10 (or 210 in the corresponding embodiment of Figure 2) is a stmctured relational 
database or the like, step 406 of the present invention may entail constructing an 

14 
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appropriate Structured Query Language (SQL) query or the like, or automatically 
filling out a firont-end query form, series of menus or the like, as described above. 

In many existing Internet (and Intranet) applications, an online electronic data 
source is accessible to users only through the medium of interaction with a so-called 
5 Common Gateway Interface (CGI) script. Typically the user who visits a web site of 
this nature must fill in the fields of an online interactive form. The online forai is in 
turn linked to a CGI script, which transparently handles actual navigation of the 
associated data source and produces output for viewing by the user's web browser. In 
other words, direct user access to the data source is not supported, only mediated 
10 access through the form and CGI script is offered. 

For applications of this nature, an advantageous embodiment of flie present 
invention "scrapes" the scripted online site where information desired by a user may 
be found in order to facilitate construction of an effective navigation query. For 
example, suppose that a user's spoken natural language request is: "What's the weather 
15 in Miami?" After this request is received at step 402 and interpreted at step 404, 
assume that step 405 determines that the desired weather information is available 
online through the medium of a CGI-scripted interactive form. Step 406 is then 
preferably carried out using the expanded process diagrammed in Figure 5. In 
particular, at sub-step 520, query constmction logic 330 electronically "scrapes" the 
online interactive form, meaning that query construction logic 330 automatically 
extracts the format and structure of input fields accepted by the online fomi. At sub- 
step 522, a navigation query is then constructed by instantiating (filling in) the 
extracted input format - essentially an electronic template ~ in a manner reflecting 
the user's request for information as interpreted in step 404. The flow of control then 
returns to step 407 of Figure 4. Ultimately, when the query thus constructed by 
scraping is used to navigate the online data source in step 408, the query effectively 
initiates the same scripted response as if a human user had visited the online site and 
had typed appropriate entries into the input fields of the online form. 

In the embodiment just described, scraping step 520 is preferably carried out 
with the assistance of an online extiraction utiUty such as WebL. WebL is a scripting 
language for automating tasks on the World Wide Web. It is an imperative. 
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interpreted language that has built-in support for common web protocols like HTTP 
and FTP, and popular data types like HTML and XML. WebL's implementation 
language is Java, and the complete source code is available from Compaq. In 
addition, step 520 is preferably perfomied dynamically when necessary - in other 
5 words, on-the-fly in response to a particular user query — but in some applications it 
maybe possible to scrape relatively stable (unchanging) web sites of likely interest in 
advance and to cache the resulting template information. 

It will be apparent, in light of the above teachings, that preferred embodiments 
of the present invention can provide a spoken natural language interface atop an 

10 existing, non-voice data navigation system, whereby users can interact by means of 
intuitive natural language input not strictly conforming to the linear browsing 
architecture or other artifacts of an existing menu/text/click navigation system. For 
example, users of an appropriate embodiment of the present invention for a video-on- 
demand application can directly speak the natural request: "Show me the movie 

15 'Unforgiven'" - instead of walking step-by-step through a typically linear sequence of 
genre/title/actor/director menus, scrolling and selecting from potentially long lists on 
each menu, or instead of being forced to use an alphanumeric keyboard that cannot be 
as comfortable to hold or use as a lightweight remote control. Similarly, users of an 
appropriate embodiment of the present invention for a web-surfing apphcation in 

20 accordance with the process shown in Figure 5 can directly speak the natural request: 
"Show me a one-month price chart for Microsoft stock" - instead of potentially 
having to navigate to an appropriate web site, search for the right ticker symbol, 
enter/select the symbol, and specify display of the desired one-month price chart, each 
of those steps potentially involving manual navigation and data entry to one or more 

25 different interaction screens. (Note that these examples are offered to illustrate some 
of tlie potential benefits offered by appropriate embodiments of the present invention, 
and not to limit the scope of the invention in any respect.) 

c. Error Correction 

Several problems can arise when attempting to perform searches based on 
30 spoken natural language input. As indicated at decision step 407 in the process of 
Figure 4, certain deficiencies may be identified during the process of query 
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construction, before search of the data source is even attempted. For example, the 
user's request may fail to specify enough infonnation in order to construct a 
navigation query that is specific enough to obtain a satisfactory search result. For 
example, a user might orally request "what's fhe weather?" whereas the national 
online data source identified in step 405 and scraped in step 520 might require 
specifying a particular city. 

Additionally, certain deficiencies and problems may arise following the 
navigational search of the data source at step 408, as indicated at decision step 409 in 
Figure 4. For example, with reference to a video-on-demand application, a user may 
Avish to see the movie "Unforgiven", but perh^s the user can't recall name of the film, 
but knows it was directed by and starred actor Clint Eastwood. A typical video-on- 
demand database might indeed be expected to allow queries specifying the name of a 
leading actor and/or director, but in the case of this query - as in many cases ~ that 
will not be enough to narrow the search to a single fihn, and additional user input in 
15 some form is required. 

In the event that one or more deficiencies in the user's spoken request, as 
processed, result in the problems described, either at step 407 or 409, some form of 
error handling is in order. A straightforward, crude technique might be for the system 
to respond simply "inptit not understood / insufficient; please try again." However, 
20 that approach will likely result in finstrated users, and is not optimal or even 
acceptable for most applications. Instead, a preferred technique in accordance with 
the present invention handles such errors and deficiencies in user mpvA at step 412, 
whether detected at step 407 or step 409, by soliciting additional input ftom the user 
in a manner taking advantage of the partial construction akeady performed and via 
user interface modaUties in addition to spoken natural language ("multi-modaUty"). 
This supplemental interaction is preferably conducted through client display device 
112 (202, in the embodiment of Figure 2), and may include textual, graphical, audio 
and/or video media. Further details and examples are provided below. Query 
refinement logic 340 preferably carries out step 412. The additional input received 
from the user is fed into and augments interpreting step 404, and query construction 
step 406 is likewise repeated with the benefit of the augmented interpretation. These 
operations, and subsequent navigation step 408, are preferably repeated until no 
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remaining problems or deficiencies are identified at decision points 407 or 409. 
Further details and examples for this query refinement process are provided 
immediately below. 

Consider again the example in which the user of a video-on-demand 
application wishes to see "Unforgiven" but can only recall that it was directed by and 
starred Clint Eastwood. First, it bears noting that using a prior art navigational 
interface, such as a conventional menu interfiice, will likely be relatively tedious in 
this case. The user can proceed through a sequence of menus, such as Gaire (select 
"western"). Title (skip). Actor ("Clint Eastwood"), and Director ("Clint Eastwood"). 
In each case -especially for the last two items ~ the user would typically scroll and 
select firom fairly long Hsts in order to enter his or her desired name, or perhaps use a 
relatively couch-unfiiendly keypad to manually type the actor's name twice. 

Using a preferred embodiment of the present invention, the user iostead speaks 
aloud, holding remote control microphone 102, "I want to see that movie starring and 
directed by Clint Eastwood. Can't remember the title." At step 402 the voice data is 
received. At step 404 the voice data is interpreted. At step 405 an appropriate online 
data source is selected (or perhaps the system is directly connected to a proprietary 
video-on-demand provider). At step 406 a query is automatically constiucted by the 
query constiiiction logic 330 specifying "Clint Eastwood" in both the actor and 
director fields. Step 407 detects no obvious problems, and so the query is 
electronically submitted and the data source is navigated at step 408, yielding a list of 
several records satisfying the query (e.g., "Unforgiven", "True Crime", "Absolute 
Power", etc.). Step 409 detects that additional user ir^ut is needed to further refine 
the query in order to select a particvdar fihn for viewing. 

At that point, in step 412 query refinement logic 340 might preferably 
generate a display for client display device 1 12 showing the (relatively short) list of 
fihn titles ttiat satisfy the user's stated constraints. The user can then preferably use a 
relatively convenient input modality, such as buttons on the remote control, to select 
the desired title firom the menu. In a fijrther prefenred embodiment, tiie first title on 
the Ust is highUghted by default, so that the user can simply press an "OK" button to 
choose that selection. In a fiirther preferred feature, the user can mix input modalities 
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by speaking a response like "I want number one on the list." Alternatively, the user 
can preferably say, "Let's see Unforgiven," having now been reminded of the tifle by 
the menu display. 

Utilizing the user's supplemental input, request processing logic 300 iterates 
again through steps 404 and 406, this time constructing a fully-specified query that 
specifically requests the Eastwood fihn "Unforgiven." Step 408 navigates the data 
source usmg that query and retrieves the desired film, which is then electronically 
transmitted in step 410 from network server 108 to client display device 112 via 
communications network 106. 

Now consider again the example in which flie user of a web surfing 
application wants to know his or her local weather, and simply asks, "what's the 
weather?" At step 402 the voice data is received. At step 404 the voice data is 
interpreted. At step 405 an online web site providing current weather information for 
major cities around the world is selected. At step 406 and sub-step 520, the onUne 
site is scraped using a Webl^style tool to extract an input template for interacting 
with the site. At sub-step 522, query construction logic 330 attempts to construct a 
navigation query by instantiating the input template, but determines (quite rightly) 
that a required field - name of city ~ cannot be determined from the user's spoken 
request as interpreted in step 404. Step 407 detects this deficiency, and in step 412 
query refinement logic 340 preferably generates output for client display device 112 
soliciting the necessary supplemental input. In a preferred embodiment, the output 
might display the name of the city where the user is located highlighted by defeult. 
The user can then simply press an "OK" button ~ or perh^s mix modalities by saying 
"yes, exactly" ~ to choose that selection. A preferred embodiment would fiirther 
display an alphabetical scrollable menu listing other major cities, and/or invite the 
user to speak or select the name of the deshed city. 

Here again, utiUzmg the user's supplemental input, request processmg logic 
300 iterates through steps 404 and 406. This time, in performing sub-step 520, a 
cached version of the input template ahready scraped in flie previous iteration might 
preferably be retiieved. In sub-step 522, query constiiiction logic 330 succeeds this 
time in instantiating the input template and constractmg an effective query, smce tiie 
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. desired city has now been clarified. Step 408 navigates the data source using that 
query and retrieves the desired weather information, which is then electronically 
transnaitted in step 410 firom network server 108 to client display device 112 via 
communications network 106- 

5 It is worth noting that in some instances, there may be details that are not 

explicitly provided by the user, but that query constmction logic 330 or query 
refinement logic 340 may preferably deduce on their own through reasonable 
assumptions, rather than requiring the use to provide explicit clarification. For 
example, in the example previously described regarding a request for a weather 

10 report, in some appHcations it might be preferable for the system to simply assume 
that the user means a weather report for his or her home area and to retrieve that 
information, if the cost of doing so is not significantly greater than the cost of asking 
the user to clarify tlie query. Making such an assxmiption might be even more 
strongly justified in a preferred embodiment, as described earHer, where user histories 

15 are tracked, and where such history indicates that a particular user or group of users 
typically expect local information when asking for a weather forecast. At any rate, in 
the event such an assimiption is made, if the user actually intended to request the 
weather for a different city, the user would tiien need to ask his or her question again. 
It will be apparent to practitioners, in light of the above teachings, that the choice of 

20 whether to program query construction logic 330 and query refinement logic 340 to 
make make particular assumptions will typically involve trade-offs involving user 
conveimce that can be assessed in the context of specific apphcations, 
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3. Open Agent Architecture (OAA®) 

Open Agent Architecture™ (OAA®) is a software platform, developed by the 
assignee of the present invention, that enables effective, dynamic collaboration among 
communities of distributed electronic agents. OAA is described in greater detail in 
5 co-pending U.S. Patent Application No. 09/225,198, which has been incorporated 
herein by reference. Very briefly, the ftinctionality of each chent agent is made 
available to the agent community through registration of the chent agent's capabilities 
with a facilitator. A software "wrapper" essentially surrovmds the underlying 
appUcation program performing the services offered by each client. The common 
10 infrastructure for constructmg agents is preferably supplied by an agent library. The 
agent library is preferably accessible in the runtime environment of several different 
programming languages. The agent Ubrary preferably minimizes the effort required 
to construct a new system and maximizes the ease with which legacy systems can be 
"wrapped" and made compatible with the agent-based architecture of the present 
15 invention. When invoked, a chent agent makes a connection to a facilitator, which is 
known as its parent facilitator. Upon comiection, an agent registers with its parent 
facihtator a specification of the capabilities and services it can provide, using a high- 
level, declarative Ihteragent Communication Language ("JCL") to express those 
capabilities. Tasks are presented to the facihtator in the form of ICL goal expressions. 
20 When a facihtator determines that the registered c^abilities of one of its cUent agents 
will help satisfy a current goal or sub-goal thereof, the facihtator delegates that sub- 
goal to the cUent agent in the form of an ICL request. The client agent processes the 
request and returns answers or information to the facihtator. In processing a request, 
the chent agent can use ICL to request services of other agents, or utilize other 
25 infrastmcture services for collaborative work. The facihtator coordmates and 
integrates the results received from different chent agents on various sub-goals, m 
order to satisfy the overall goal. 

OAA provides a useful software platform for building systems that integrate 
spoken natural language as well as other user mput modalities. For example, see the 
30 above-referenced co-pending patent application, especially Figure 13 and the 
corresponding discussion of a "multi-modal maps" appUcation, and Figure 12 and the 
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corresponding discussion of a "xmified messaging" supplication. Another example is 
the hifoWiz interactive information kiosk developed by the assignee and described in 
the docimient entitled "InfoWiz: An Animated Voice Interactive iafonnation System" 
available online at httD://www.ai.sri.com/-oaa/apphcations.html , A copy of the 
Info Whiz document is provided in an Information Disclosure Statement submitted 
herewith and incorporated herein by this reference. A further example is the 
"CommandTalk" apphcation developed by the assignee for the U.S. military, as 
described online at http://www.ai.sri.com/-lesa£^commandtalk.html and in the 
following publications, copies of which are provided in an Information Disclosure 
Statement submitted herewith and incorporated herein by this reference: 

• "CommandTalk: A Spoken-Language Interface 
for Battlefield Simulations", 1997, by Robert Moore, John Dowding, Harry 
Bratt, J. Mark Gawron, Yonael Gorfii and Adam Cheyer, in "Proceedings of 
the Fifth Conference on Applied Natural Language Processmg", Washington, 
DC, pp. 1-7, Association for Computational Linguistics 

• "The CommandTalk Spoken Dialogue System", 1999, by 
Amanda Stent, John Dowding, Jean Mark Gawron, Elizabeth Owen Bratt and 
Robert Moore, in "Proceedings of the Thirty-Seventh Annual Meeting of the 
ACL", pp. 183-190, University of Maryland, College Park, MD, Association 
for Computational Linguistics 

"Interpreting Language in Context in ConmiandTalk", 1 999, by 
John Dowding and Elizabeth Owen Bratt and Sharon Goldwater, in 
"Communicative Agents: The Use of Natural Language in Embodied 
Systems", pp. 63-67, Association for Computing Machinery (ACM) Special 
Interest Group on Artificial Intelligence (SIGART), Seattle, WA 

For some applications and systems, OAA can provide an advantageous 
platform for constructing embodiments of the present invention. For example, a 
representative application is now briefly presented, with reference to Figure 6. If the 
statement "show me movies starring John Wayne" is spoken into the voice input 
device, the voice data for this request will be sent by UI agent 650 to facilitator 600, 
which in turn will ask natural language (NL) agent 620 and speech recognition agent 
610 to interpret the query and return the interpretation in ICL format. The resulting 
ICL goal expression is then routed by the facihtator to appropriate agents - in this 
case, video-on-demand database agent 640 - to execute the request. Video database 
agent 640 preferably includes or is coupled to an appropriate embodiment of query 
construction logic 330 and query refinement logic 340, and may also issue ICL 

22 



0169449A2_L> 



wo 01/69449 PCT/USOl/07924 

requests to facilitator 600 for additional assistance - e.g., display of menus and 
capture of additional user input in the event that query refinement is needed - and 
facilitator 600 will delegate such requests to appropriate chent agents in the 
community. When the desired video contCTit is ultimately retrieved by video database 
5 agent 640, UI agent 650 is invoked by faciUtator 600 to display the movie. 

Other spoken user requests, such as a request for the current weather in New 
York City or for a stock quote, would eventually lead facilitator to invoke web 
database agent 630 to access the desired infonnation from an appropriate Internet site. 
Here again, web database agent 630 preferably includes or is coupled to an 

10 appropriate embodiment of query construction logic 330 and query refinement logic 
340, including a scraping utility such as WebL. Other spoken requests, such as a 
request to view recent emails or access voice mail, would lead the faciUtator to mvoke 
the appropriate email agent 660 and/or telephone agent 680. A request to record a 
televised program of interest might lead facilitator 600 to invoke web database agent 

15 630 to return televised program schedule infonnation, and then invoke VCR 
controller agent 680 to program the associated VCR unit to record the desired 
television program at the schedided time. 

Control and connectivity embracing additional electronic home appUances 
(e.g., microwave oven, home surveillance system, etc.) can be integrated in 

20 comparable fashion. Indeed, an advantage of OAA-based embodiments of the present 
invention, that will be apparent to practitioners in light of the above teachings and in 
light of the teachings disclosed in the cited co-pending patent appUcations, is the 
relative ease and flexibiUty with which additional service agents can be plugged into 
the existing platform, immediately enabling the faciUtator to respond dynamically to 

25 spoken natural language requests for the corresponding services. 
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4. Further Embodiments and Equivalents 

While the present invention has been described in terms of several preferred 
embodiments, there are many alterations, permutations, and equivalents that may fall 
within the scope of this invention. It should also be noted that there are many 
alternative ways of implementing the methods and apparatuses of the present 
invention. It is therefore intended that the following appended claims be interpreted 
as including all such alterations, permutations, and equivalents as fall within the true 
spirit and scope of the present invention. 
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Claims 

WJiat is claimed is: 

1 . A method for speech-based navigation of an electronic data source, the 
electronic data source having a scripted online interface and being located at one or 
more network servers located remotely from a user, comprising the steps of: 

(a) receiving a spoken request for desired information from the user; 

(b) rendering an interpretation of the spoken request; 

(c) dynamically scraping the onhne scripted interface to the data source, to 
5 extract an input template; 

(d) constractmg a navigation query based upon the interpretation of step 
(b) and the template of step (c); 

(e) using the navigation query to select a portion of the electronic data 
source; and 

(0 transmitting flie selected portion of the electronic data source from the 
network server to a cUent device of the user. 

2. The method of claim 1, wherein the online scripted interface is a web site. 

3. The method of claim 1, wherein the input template is extracted from an online 
fomi of the scripted interface. 

15 4. The method of claim 1 , wherein the step of rendermg an interpretation further 
includes deriving Unguistic information by usmg a speech recognition engine and a 
Unguistic parser. 



5. The method of claim 1 , wherein at least one of the steps of rendering 
interpretation and constmcting a navigation query are perfomied, at least in part, 
20 computing device located locally with the user. 



an 

on a 
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6. The method of claim 1 , wherein at least one of the steps of rendering an 
interpretation and constructing a navigation query are performed, at least in part, on a 
network computing device located remotely from the user. 

7. The method of claim 1 , further comprising the step of soliciting additional 
5 input in response to one or more dejSciencics encountered during the step of 

constructing the navigation query. 

8. The method of claun 7, wherein the deficiencies include unresolved words of 
the spoken request. 

9. The method of claim 7, wherein tlie deficiencies include one or more required 
10 elements of the navigational query not detenninable from the interpretation of the 

spoken request 

10. The method of claim 7, wherein the step of soliciting additional input is 
performed in response to one or more deficiencies encountered after a first navigation 
of tlae data source using the navigation query constructed in step (d). 

15 11. The method of claim 10, wherein the deficiencies include existence of more 
than one data record within the data source responsive to the navigation query. 

12. The method of claim 10, wherein the deficiencies include failure to identify a 
single data record within the data source responsive to the navigation query. 

13. The method of claim 7, further comprising the steps of: 

(a) receiving the additional input from the user 

(b) refining the navigational query based on the additional input; 

(c) detemiining whether the navigational query is deemed 
adequate; 

(d) soliciting additional input if the navigational query is not 
deemed adequate and repeating steps (a)-(c) until the 
navigational query is deemed adeqixate. 
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1 4. The method of claim 7, wherein a determination that deficiencies exist in the 
navigational query is at least partially based on a user-input statement that additional 
information is required, 

15. The method of claim 7, wherein the additional input is soUcited by presenting 
5 a menu to the user on the client device of the user. 

16. The method of claim 7, wherein the additional input is solicited by presenting 
a textual request for the additional input 

17. The method of claim 7, wherein the additional input is solicited by outputting 
an audible request for the additional input. 

10 18. The method of claim 7, wherein the additional input is solicited by presenting 
a list of portions of the electronic data source that match the navigational query. 

1 9. The method of claim 7, wherein the additional input received from the user is 
at least partially speech based. 

20. The method of claim 7, wherein the additional input received from the user 
15 includes no spoken input. 

2 1 . The method of claim 1 , wherein the method is performed with respect to a 
plurality of simultaneous users and corresponding cUent devices. 

22. The method of claim 1, fiirther including the step of selecting the data source 
from among a plurality of candidate electronic data sources, in response to the 

20 interpretation of the spoken request. 

23. The method of claim 1, wherein at least a portion of a communications hnk 
between the client device and the data source is the Intemet. 

24. A system for speech-based navigation of an electronic data source, the 
electronic data source having a scripted onUne interface and bemg located at one or 

25 more network servers located remotely from a user, the system comprising: 

(a) a portable microphone operable to receive a spoken request for desired 
information from the user; 
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(b) language processing logic, operable to render an interpretation of the 
spoken request; 

(c) template extraction logic, operable to dynamically scrape the online 
scripted interface to the data source, to extract an input template; 

5 (d) query construction logic, operable to construct a navigation query 

based upon the interpretation rendered by the language processing 
logic and the template extracted by the template extraction logic; 

(e) navigation logic, operable to select a portion of the electronic data 
source using the navigation query; and 

(0 electronic commimications infrastructure for transmitting the selected 
portion of the electronic data source from the network server to a client 
device of the user. 

25. The system of claim 24, wherein the online scripted interface is a web site. 

26. The system of claim 24, wherein the input template is extracted from an online 
15 form of the scripted interface. 

27. The system of claim 24, wherein the language processing logic includes 
speech recognition logic and linguistic parsing logic for deriving linguistic 
information, 

28. The system of claim 24, wherein at least a portion of the language processing 
20 logic is hosted on a computing device located locally with the user. 

29. A computer program embodied on a computer readable medium for utilizing 
spoken natural language for navigating an electronic data source, the electronic data 
source having a scripted onUne interface and being located at one or more network 
servers located remotely from a user, comprising: 

^ c^d® segment that receives a spoken natural language ("NL") request 
for desired information from the user; 
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(b) a code segment that renders an interpretation of the spoken natural 
language request; 

(c) a code segment that dynamically scrapes the online scripted interface 
to the data source, to extract an input template; 

5 (d) a code segment that constructing a navigation query based upon the 

interpretation of step (b) and the template of step (c); 

(e) a code segment that uses the navigation query to select a portion of the 
electronic data source; anda code segment that transmits the selected 
portion of the electronic data source from the network server to a client 
10 device of the user. 

30. The system of claim 24, further comprising user interaction logic for soliciting 
additional input in response to one or more deficiencies encountered during the 
constructing the navigation query. 

3 1 . The system of claim 30, wherein the deficiencies include unresolved words of 
1 5 the spoken request. 

32. The system of claim 30, wherein the deficiencies include one or more required 
elements of the navigational query not determinable from the interpretation of the 
spoken request. 



20 



25 



33. The system of claim 30, wherein the user interaction logic solicits additional 
input in response to one or more deficiencies encountered after a first navigation of 
the data source using the navigation query constructed by the query construction 
logic. 

34. The system of claim 33, wherein the deficiencies include existence of more 
than one data record within the data source responsive to the navigation query. 

35. The system of claim 33, wherein the deficiencies include failure to identiiy a 
single data record within the data source responsive to the navigation query. 
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36. The system of claim 24 wherein the navigation logic selects the data source 
from among a plurality of candidate electronic data sources, in response to the 
interpretation of the spoken request. 

37. The system of claim 24, wherein at least a portion of the electronic 
5 communications infrastructure is the Mtemet. 

38. A computer program embodied on a computer readable medium for speech- 
based navigation of an electronic data source, the electronic data source having a 
scripted online interface and being located at one or more network servers located 
remotely from a user, comprising: 

(a) a code segment that receives a spoken request for desired information 
from the user; 

(b) a code segment that renders an interpretation of the spoken request; 

(c) a code segment that dynamically scrapes the online scripted interface 
to the data source, to extract an input template; 

(^) a <^ode segment that constructing a navigation query based upon the 
interpretation of step (b) and the template of step (c); 

(e) a code segment that uses the navigation query to select a portion of the 
electronic data source; and 

(f) a code segment that transmits the selected portion of the electronic data 
sowce from the network server to a client device of the user. 

39. The computer program of claim 38, wherein the online scripted interface is a 
web site. 

40. The computer program of claim 38, wherem the input template is extracted 
from an online form of the scripted interface. 

25 41 . The computer program of claim 38, further comprising a code segment that 
derives linguistic information by using a speech recognition engine and a Unguistic 
parser. 



30 



BNSDOCID: <WO 0169449A2J_> 



wo 01/69449 



PCT/USOl/07924 



42. The computer program of claim 38, wherein the rendering of the interpretation 
and construction of the navigation query are performed, at least in part, on a 
computing device located locally with the user. 

43. The computer program of claun 38, wherein the rendering of the interpretation 
5 and construccion of the navigation query are performed, at least in part, on a network 

computing device located remotely from the user. 

44. The computer program of claim 38, further comprising a code segment tliat 
sohcits additional input in response to one or more deficiencies encountered during 
construction of the navigation query. 

10 45. The computer program of claim 45, wherein the deficiencies include 
unresolved words of the spoken request. 

46. The computer program of claun 45, wherein the deficiencies include one or 
more required elements of the navigational query not deteiminable from the 
interpretation of the spoken request. 

15 47. The computer program of claim 45, wherein the code segment that sohcits 
additional input sohcits the additional input in response to one or more deficiencies 
encountered after a first navigation of the data source using the navigation query. 

48. The computer program of claim 47, wherein the deficiencies include existence 
of more than one data record within the data source responsive to the navigation 
20 query. 

40. The computer program of claim 47, wherein the deficiencies include failure to 
identify a single data record within the data source responsive to the navigation query. 

41 . The computer program of claim 44, fijrther comprising the steps of: 

(a) a code segment that receives the additional input from the user 



25 



(b) a code segment that refines the navigational query based on the 
additional input; 
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(c) a code segment that determines whether the navigational query 
is deemed adequate; 

(d) a code segment that solicits additional input if the navigational 
query is not deemed adequate and repeating steps (a)-(c) until 

5 the navigational query is deemed adequate. 

42. The computer program of claim 44, wherein a detemiination that deficiencies 
exist in the navigational query is at least partially based on a user-input statement that 
additional information is required, 

43. The computer program of claim 44, wherein the additional input is sohcited 
10 by presenting a menu to the user on the cUent device of the user. 

44. The computer program of claim 44, wherein the additional input is solicited 
by presenting a textual request for the additional input. 

45. The computer program of claim 44, wherein the additional input is sohcited 
by outputting an audible request for the additional input. 

15 46. The computer program of claim 44, wherein the additional input is sohcited 
by presenting a hst of portions of the electronic data source that match the 
navigational query. 

47. The computer program of claim 44, wherein the additional input received 
from the user is at least partially speech based. 

20 48. The computer program of claim 44, wherein the additional mput received from 
the user includes no spoken input. 

49. The computer program of claim 38, wherem the code segments of the 
computer program operate with respect to a plurality of simultaneous users and 
coixesponding client devices. 

25 50. The computer program of claim 38, further including a code segment that 

selects the data source from among a plurality of candidate electronic data sources, in 
response to the interpretation of the spoken request. 
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5 1 . The computer program of claim 38, wherein at least a portion of a 
communications link between the client device and the data source is the Intemet, 

52. A metliod for utilizing spoken natural language for navigating an electronic 
data source, the electronic data source having a scripted online interface and being 
located at one or more network servers located remotely from a user, comprising the 
steps of: 

(a) receiving a spoken natural language ("NL") request for desired 
information from the user; 

(b) rendering an interpretation of the spoken natural language request; 

(c) dynamically scrapmg the online scripted interface to the data source, to 
extract an input template; 

(d) constructing a navigation query based upon the interpretation of step 
(b) and the template of step (c); 

(e) using the navigation query to select a portion of the electronic data 
source; and 

(£) transmitting the selected portion of the electronic data source from the 
network server to a chent device of the user. 

53. The method of claim 52, wherein the online scripted interface is a web site. 

54. The method of claim 52, wherein the input template is extracted from an 
online form of the scripted interface. 

55. The method of claim 52, wherein the step of rendering an interpretation 
fiirther mcludes deriving linguistic information by using a speech recognition engine 
and an NL parsCT. 



56. The method of claim 52, wherein at least one of the steps of rendering 
interpretation and constructing a navigation query are perforated, at least in part, 
computing device located locally with the usen 



an 
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57. The method of claim 52, wherein at least one of tiie steps of rendering an 
interpretation and constructing a navigation query are performed, at least in part, on a 
network computing device located remotely from the user. 

58. The method of claim 52, fiirther comprising the step of soliciting additional 
5 input in response to one or more deficiencies encountered durnig the step of 

constructing the navigation query. 

59. The method of claim 58, wherein the deficiencies include unresolved words 
of the spoken NL request. 

60. The method of claim 5 8, wherein the deficiencies include one or more 

10 required elements of the navigational query not determinable from the interpretation 
of the spoken NL request. 

61 . The method of claim 58, wherein the step of soliciting additional input is 
performed in response to one or more deficiencies encountered afl:er a first navigation 
of the data source using the navigation query constmcted in step (d). 

15 62. The method of claim 61, wherein the deficiencies include existence of more 
than one data record within the data source responsive to the navigation query. 

63. The method of claim 61, wherein the deficiencies include failure to identify a 
single data record wifliin the data source responsive to the navigation query, 

64. The method of claim 52, wherein the method is performed with respect to a 
20 plurahty of simultaneous users and corresponding client devices. 

65. The method of claim 52, fiirther including the step of selecting the data source 
from among a plurality of candidate electronic data sources, in response to the 
interpretation of the spoken NL request. 

66. The method of claim 52, wherein at least a portion of a communications link 
25 between the chent device and the data source is the Internet. 

67. A system for utilizing spoken natural language for navigating an electronic 
data source, the electronic data source having a scripted online interface and being 
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located at one or more network servers located remotely from a user, the system 
comprising: 

(a) a portable microphone operable to receive a spoken natural language 
("NL") request for desired information from the user; 

5 (b) spoken language processing logic, operable to render an interpretation 

of the spoken natural language request; 

(c) template extraction logic, operable to dynamically scrape the online 
scripted interface to the data source, to extract an input template; 

(d) query construction logic, operable to construct a navigation query 
^ ^ based upon the interpretation rendered by the spoken language 

processing logic and the template extracted by the template extraction 
logic; 

(e) navigation logic, operable to select a portion of the electronic data 
source using the navigation query; and 

(0 electronic communications infrastructure for transmitting the selected 
portion of the electronic data source from the network server to a client 
device of flie user. 

68. The system of claim 67, wherein the online scripted interface is a web site. 

69. The system of claim 67, wherein the input template is extracted from an 
20 online fomi of the scripted interface. 

70. The system of claim 67, wherein the spoken language processing logic 
includes speech recognition logic and an NL parsing logic for deriving linguistic 
information. 

71 . The system of claim 67, wherein at least a portion of the spoken language 
25 processing logic is hosted on a computing device located locally with tlie user. 
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72. The system of claim 67, further comprising user interaction logic for 
soliciting additional input in response to one or more deficiencies encountered during 
the constructing the navigation query. 

73. The system of claim 72, wherein the deficiencies include unresolved words of 
5 the spoken ML request. 

74. The system of claim 72, wherein the deficiencies include one or more 
required elements of the navigational query not determinable fi-om the interpretation 
of the spoken NL request. 

75 . The system of claim 72, wherein the user interaction logic sohcits additional 
10 input in response to one or more deficiencies encountered after a first navigation of 

the data source using the navigation query constructed by the query constraction 
logic. 

76. The system of claim 75, wherein the deficiencies include existence of more 
than one data record within the data source responsive to the navigation query. 

15 77. The system of claim, 75, wherein the deficiencies include failure to identify a 
single data record within the data source responsive to the navigation query. 

78. The system of claim 67, wherein the navigation logic selects the data source 
from among a plurality of candidate electronic data sources, in response to the 
interpretation of the spoken NL request. 

20 79. The system of claim 67, wherein at least a portion of the electronic 
communications infrastructure is the Intemet. 

80. A computer program embodied on a computer readable medium for utilizing 
spoken natural language for navigating an electronic data source, the electronic data 
source having a scripted online interface and being located at one or more network 
25 servers located remotely from a user, comprising: 

(f) a code segment that receives a spoken natural language ('•NL'O request 
for desired information from the user; 
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(g) a code segment that renders an interpretation of the spoken natural 
language request; 

(h) a code segment that dynamically scrapes the online scripted interface 
to the data source, to extract an input template; 

5 (i) a code segment that constructing a navigation query hased upon the 

interpretation of step (b) and the template of step (c); 

0) a code segment that uses the navigation query to select a portion of the 
electronic data soiu-ce; and 

(k) a code segment fliat transmits the selected portion of the electronic data 
source from the network server to a cUent device of the user. 

8 1 . The computer program of claim 80, wherein the online scripted interface is a 
web site. 

82. The computer program of claim 80, wherein the input template is extracted 
from an online form of the scripted interface. 

15 83. The computer program of claim 80, further comprising a code segment that 
derives linguistic information by using a speech recognition engine and an NL parser. 

84. The computer program of claim 80, wherein the rendering of the 
interpretation and constraction of the navigation query are performed, at least in part, 
on a computing device located locally with the user. 

20 85. The computer program of claim 80, wherein the rendering of the 

interpretation and construction of the navigation query are performed, at least in part, 
on a network computing device located remotely from the user. 

86. The computer program of claim 80, fiarther comprising a code segment that 
solicits additional input in response to one or more deficiencies encountered during 

25 construction of the navigation query. 

87. The computer program of claim 86, wherein the deficiencies include 
unresolved words of the spoken NL request. 
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88. The computer program of claim 86, wherein the deficiencies include one or 
more required elements of the navigational query not determinable from the 
interpretation of the spoken NL request. 

89. The computer program of claim 86, wherein tiie code segment tliat soUcits 
5 additional input solicits the additional input in response to one or more deficiencies 

encountered after a first navigation of the data source using the navigation query. 

90. The computer program of claim 89, wherein the deficiencies include 
existence of more than one data record within the data source responsive to the 
navigation query. 

10 91 . The computer program of claim 89, wherein the deficiencies include failure to 
identify a single data record within the data source responsive to the navigation query. 

92. The computer program of claim 80, wherein the code segments of the 
computer program operate with respect to a plurality of simultaneous users and 
corresponding client devices. 

15 93. The computer program of claim 80, further including a code segment that 

selects the data source from among a plurality of candidate electronic data sources, in 
response to the interpretation of the spoken NL request. 

94. The computer program of claim 80, wherein at least a portion of a 
communications link between the chent device and the data source is the Internet. 

20 
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